The NOAA Virtual Data System: Improving Access to and Management of Federal Environmental Data

The National Oceanic & Atmospheric Administration’s (NOAA) National Environmental Satellite, Data & Information Service (NESDIS) operates three national data centers that are geographically dispersed and provide multidisciplinary environmental data and information to researchers, government planners, business decision makers, lawyers and other professionals, teachers, students, and the general public. At each of these centers there are multiple formats of data stored on a variety of mass storage devices. Data users must cope with numerous access systems related to the specific type of data and primary customers. An individual familiar with these centers who requires only a single type of data can obtain what he/she needs in a reasonable amount of time. It has become essential for NOAA to change the way it manages these data because of (1) an exponential increase in the volume of data archived, (2) a growing demand for the ability to combine multiple types of environmental data, and (3) customer expectation of instant response to data requests. For many years NOAA’s management had a vision of a single unified system to more effectively manage these valuable data and to provide faster and easier customer access. In 1996 funds were appropriated to start development of this system. A new, virtual system is now being implemented to pull these diverse data center operations together. Customers can browse across the distributed data sub-systems and place one order for geophysical, oceanographic, and climatic data, independent of data location, format or storage media. To put these concepts into an operational system with a small budget requires judicious allocation of resources. Numerous storage technologies are being integrated and connected through a scaleable Virtual Private Network (VPN). Other major subsystems of the NOAA Virtual Data System (NVDS) project consist of (1) systems performance monitoring, (2) customer and order management processing, (3) data storage infrastructure, (4) security, (5) metadata management, and (6) customer access mechanisms. Standards Based Architecture (SBA) is being used in developing the NVDS. Maximum flexibility was a requirement from the outset of the project to be able to quickly adapt to the latest available hardware and software systems and get the biggest payoff from the limited funds available. The system must pull together numerous data sets that vary in size from a few megabytes to tens of terabytes. These data currently exist in thousands of formats and include points, track lines, time series, grids, and images. NOAA management was determined to reduce costs by making use of the maximum amount of COTS (Commercial Off The Shelf) software, implementing standards and minimizing the amount of unique software to be developed and maintained.